Method and system for processing a plurality of images so as to detect lanes on a road

ABSTRACT

A system and a method for processing a plurality of images, each image of the plurality of images being acquired by a respective image acquisition module of a vehicle and each image acquisition module being oriented outwardly with respect to the vehicle, the method comprising:elaborating a bird&#39;s eye view image of surroundings of the vehicle using pixel values of pixels of at least one portion of each image of the plurality of images as pixel values of the bird&#39;s eye view image, andperforming, on the bird&#39;s eye view image, a detection of at least one lane marked on a surface on which the vehicle is and visible on the bird&#39;s eye view image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a National Stage of International ApplicationPCT/EP2019/072933 filed on Aug. 28, 2019, the entire contents of whichare hereby incorporated by reference

FIELD OF THE DISCLOSURE

The present disclosure is related to the field of image processing andto the detection of objects on an image, particularly lanes marked on aroad.

DESCRIPTION OF THE RELATED ART

It is particularly desirable to perform detection of the lanes marked onroads on which vehicles are travelling, as this information is usefulfor assisting drivers or in automated driving. To this end, many lanedetection methods have been proposed to be used in automotiveapplications such as lane keep assist, lane departure detection, orparking assist.

For example, the patent application US 2017/0068862 discloses a methodthat performs lane detection on the basis of a single image.

Similar methods can be used for forward traffic scene analysis,collision warning, and these methods typically use a single cameramounted in the vicinity of a vehicle's windshield. These methods havebeen proposed to perform detection of the two lanes surrounding avehicle on a road (often called by the person skilled in the art asego-lanes) or multiple lanes. These methods do not provide enoughinformation because they only use a portion of the surroundings of thevehicle.

Other methods have proposed to detect lanes all around a vehicle, in a360° manner, often called 360 surround lane detection. For example,document U.S. Pat. No. 9,056,630 discloses the use of a bird's eye viewimage (also called top view) which represents the surroundings of thevehicle, but, in this method, the immediate surroundings is limited to1.5 to 3 meters.

In another example, patent document WO 2016013174 from Denso, a lanedetection device is described, equipped with side and front detectionunits to detect, correct or compare lane division lines ahead and on theside. The method of this document is focused on ego-lane positioning.

These solutions cannot be used and are not satisfactory for applicationsother than maintaining a vehicle in its ego-lanes.

Among the obstacles preventing obtaining bird's-eye view images coveringa greater zone, the person skilled in the art faces the wide angle ofthe cameras used, and the resolution of the cameras.

Known methods are often based on the visibility and the detection of thelane markers, they therefore require sufficient amounts of contrast (orgradient information) between markers and road area. These known methodsmay fail when the lane marker contrast is too low or when markers arefar apart or occluded (for example by traffic) or missing.

From the prior art, document “Towards End-to-End Lane Detection: anInstance Segmentation Approach” (Davy Neven, Bert De Brabandere,Stamatios Georgoulis, Marc Proesmans, Luc Van Gool) is also known.

SUMMARY OF THE DISCLOSURE

The present disclosure overcomes one or more deficiencies of the priorart by proposing a method for processing a plurality of images, eachimage of the plurality of images being acquired by a respective imageacquisition module of a vehicle and each image acquisition module beingoriented outwardly with respect to the vehicle, the method comprising:

-   -   elaborating a bird's eye view image of the surroundings of the        vehicle using the pixel values of the pixels of at least one        portion of each image of the plurality of images as pixel values        of pixels of the bird's eye view image,    -   performing, on the bird's eye view image, a detection of at        least one lane marked on the surface on which the vehicle is and        visible on the bird's eye view image.

The images from the set of images may be RGB (Red-Green-Blue) imagesobtained by said image acquisition modules, for example cameras.

A bird's eye view refers to a view from the top of the vehicle, forexample the view that a camera hovering over the vehicle and directed atthe vehicle could capture.

The inventors of the present disclosure have observed that it ispossible to equip a vehicle with a plurality of cameras and to use theimages from these cameras to combine them into a bird's eye view imageby directly using the pixel values of the pixels of the images, whichprovide simple and fast combination of the images into a single imagewhich can show the surroundings within a distance of at least ten metersand not just 1.5 to 3 meters (and therefore including all the lanes of aroad). More precisely, the portion of each image which corresponds tothe road can be used for a portion of the bird's eye view image whichshows portions of lanes around the vehicle.

Also, it is easier to perform detection of lanes marked on a road in thebird's eye view image rather than on a plurality of images.

According to a particular embodiment, the detection of lanes isperformed using a neural network.

The inventors of the present disclosure have observed that neuralnetwork (a deep learning approach) can be used for the detection oflanes on bird's eye view images with the following advantages withrespect to traditional non-deep learning approaches:

-   -   neural networks can better cope with differences in sharpness        around the lanes marked on roads (on images directly obtained        from cameras, distant lanes are smeared due to the limitations        in term of resolution);    -   neural networks can handle occlusions of marked lanes by        surrounding vehicles or by other structures, because neural        networks can take into account the context of an entire image;        thus, even if lanes partially appear or appear in an imperfect        manner, a neural network can predict their position;    -   neural networks do not assume that lanes marked on a road are        supposed to be straight, curved lanes, splits, or merges can be        dealt with if there is sufficient data to train the neural        network.

According to a particular embodiment, elaborating a bird's eye viewimage of the surroundings of the vehicle comprises using at least onelook-up table to associate the pixels of said portions of the image ofthe plurality of images to pixels of the bird's eye view image.

Using at least one look-up table is a simple implementation of using thepixel values of the pixels of at least one portion of each image of theplurality of images as pixel values of pixels of the bird's eye viewimage.

According to a particular embodiment, elaborating a bird's eye viewimage of the surroundings of the vehicle comprises using a look-up tablefor each image of the plurality of image.

According to a particular embodiment, the method comprises obtainingposition information of the vehicle from at least one sensor differentfrom a camera and taking this position information into account whenelaborating the bird's eye view image.

For example, a distance sensor may be used to determine the position ofthe vehicle with respect to the ground, for example a laser sensor suchas a lidar (solid state or mechanical).

According to a particular embodiment, the method comprises performing,on the bird's eye view image, a detection of a plurality of lanes markedon the surface on which the vehicle is and visible on the bird's eyeview image.

It should be noted that this detection may be performed using a singleneural network.

For example, this neural network may assign to each pixel in the bird'seye view a label indicating whether this pixel belongs to a lane or not.

Additionally, the neural network may be configured to identify separatelanes as different instances. For example, each lane may be assigned adifferent identifier and the neural network may assign labels.

According to a particular embodiment, the method comprises a furtherpost-processing step in which a curve is fitted on each detected lane,and a tracking step of the curve with respect to a previously obtainedplurality of images.

According to a particular embodiment, fitting a curve on each detectedlane is performed using a multi-curve fitting approach.

Multi-curve fitting approaches comprise fitting a plurality of curvesquasi simultaneously, and obeying some geometrical constraints betweenthe individual curves, for example a plurality of curves being parallelto one another and equidistant.

It should be noted that fitting a curve may comprise determining a curvethat minimizes the distance between the pixels of a lane and a curvedefined using a mathematical representation of a curve, typically apolynomial expression of a given order (second or third) or a spline(for example a piecewise spline).

Consequentially, tracking may be performed by subjecting the parametersof the curve (for example the polynomial values) to a process such as aKalman filter or an extended Kalman filter.

According to a particular embodiment, the method comprises a preliminarycalibration step to obtain calibration data in which pixels of the atleast one portion of each image are associated with pixels of the bird'seye view image.

This calibration data contains information such as pairs of pixellocations (a pixel location of a pixel from an image of the plurality ofimage and a pixel location of a pixel in the bird's eye view image.

Also, the calibration step may comprise obtaining the intrinsic andextrinsic parameters of each image acquisition module. The personskilled in the art will know how to obtain these parameters, theintrinsic parameters being parameters that are internal and fixed to aparticular image acquisition module and the extrinsic parameters beingparameters that are external to the image acquisition module and whichdepend on the position and orientation of the image acquisition module.

By way of example, the calibration step may comprise elaborating the atleast one look-up table (for example using the extrinsic and intrinsicparameters.

Also, by way of example, the extrinsic and intrinsic parameters may beobtained automatically through:

-   -   predefined measurements and/or patterns with known 3D position,        which may be called short range measurements, and through    -   longer range measurements such as the ones described in document        “Structure from Motion & Camera Self-Calibration” (Kassem Al        Ismaeil, Université de Bourgogne), document “Multiple view        geometry in computer vision” (Richard Hartley and Andrew        Zisserman, Cambridge University Press, 2004).

Also, the calibration step may comprise receiving commands inputted by auser on a human machine interface.

The present disclosure also proposes a system for processing a pluralityof images, each image of the plurality of images having been acquired bya respective image acquisition module of a vehicle and each imageacquisition module being oriented outwardly with respect to the vehicle,the system comprising:

-   -   a module configured to elaborate a bird's eye view image of the        surroundings of the vehicle using the pixel values of the pixels        of at least one portion of each image of the plurality of images        as pixel values of the bird's eye view image,    -   a module configured to perform, on the bird's eye view image, a        detection of at least one lane marked on the surface on which        the vehicle is and visible on the bird's eye view image.

This system may be configured to perform all the steps of the method asdefined above.

The present disclosure also proposes a vehicle comprising the abovedefined system and equipped with said image acquisition modules.

In one particular embodiment, the steps of the method for processing animage are determined by computer program instructions.

Consequently, the present disclosure is also directed to a computerprogram for executing the steps of a method as described above when thisprogram is executed by a computer.

This program can use any programming language and take the form ofsource code, object code or a code intermediate between source code andobject code, such as a partially compiled form, or any other desirableform.

The present disclosure is also directed to a computer-readableinformation medium containing instructions of a computer program asdescribed above.

The information medium can be any entity or device capable of storingthe program. For example, the medium can include storage means such as aROM, for example a CD ROM or a microelectronic circuit ROM, or magneticstorage means, for example a diskette (floppy disk) or a hard disk.

Alternatively, the information medium can be an integrated circuit inwhich the program is incorporated, the circuit being adapted to executethe method in question or to be used in its execution.

BRIEF DESCRIPTION OF THE DRAWINGS

How the present disclosure may be put into effect will now be describedby way of example with reference to the appended drawings, in which:

FIG. 1 is a block diagram of an exemplary method for processing aplurality of images according to an example,

FIG. 2A is a graphical representation showing the elaboration of thebird's eye view image,

FIG. 2B is another graphical representation showing the elaboration ofthe bird's eye view image,

FIG. 2C is yet another graphical representation showing the elaborationof the bird's eye view image,

FIG. 3A shows images obtained by cameras on a vehicle and thecorresponding bird's eye view,

FIG. 3B shows another image obtained by cameras on a vehicle and thecorresponding bird's eye view,

FIG. 4A shows a bird's eye view and a graphical representation of thelanes marked on the road and their detection,

FIG. 4b shows another bird's eye view and a graphical representation ofthe lanes marked on the road and their detection,

FIG. 4C shows yet another bird's eye view and a graphical representationof the lanes marked on the road and their detection,

FIG. 4D shows still another bird's eye view and a graphicalrepresentation of the lanes marked on the road and their detection, and

FIG. 5 is a schematic representation of a vehicle according to anexample.

DETAILED DESCRIPTION

An exemplary method processing a plurality of images will be describedhereinafter, as well as a corresponding system embedded on a vehicle.

A method for processing an image is represented on FIG. 1. This methodis able to detect a plurality of lanes marked on a road which arevisible or at least partially visible on a plurality of images acquiredin step S01 by a plurality of image acquisition module (typically acamera) of a vehicle circulating on a road. For example, step S01 maycomprise the acquisition of an image taken from the front of thevehicle, an image taken from the back of the vehicle, and images takenfrom both lateral sides of the vehicle. The person skilled in the artwill know how to place cameras around a vehicle so that each imageacquired shows at least a portion of the road. Also, in this example,all the cameras are oriented outwardly with respect to the vehicle(their field of view is oriented outwardly).

In step S02, for an image acquired in step S01, a look-up table is usedwhich receives this image and outputs the pixel values (for example RedGreen Blue values) for a portion of the image, typically the portion onwhich the road should be seen on the image.

It should be noted that the present disclosure is not limited to the useof look-up tables but also concerns other means to elaborate bird's eyeview images using the pixel values of the pixels of at least one portionof each image of the plurality of images as pixel values of pixels ofthe bird's eye view image.

In order to perform step S02 a preliminary step PS01 can be performed soas to determine which pixel positions of the images of a cameracorrespond to possible pixel positions in a bird's eye view image. Byway of example, this preliminary step is a calibration step performed byan operator, or performed automatically using a detection method (forexample an image segmentation method known by the person skilled in theart).

For another image acquired in step S01, a step S02′ identical to thepreviously described S02 is performed. A different calibration isperformed for this camera in the preliminary step PS01′. The calibrationstep is different in that it concerns a camera with a different field ofview or viewing angle; for example, one could show an image taken fromthe back of the vehicle while the other could show an image taken fromthe front of the vehicle.

Additional steps may be performed for other images acquired in step S01.It should be noted that steps S02, S02′, PS01, and PS01′ may be affectedby a position information of the vehicle from at least one sensordifferent from a camera (for example a distance sensor) so that thisposition information is taken into account in these steps.

For example, the distance between the ground and the vehicle may varyand can lead to applying an offset in the look-up tables.

In step S03, the output of each look-up stable is assembled, so that instep S04, a bird's eye view image is elaborated. In this image, eachpixel in a portion of each image acquired in step S01 is associated inat least a portion of the bird's eye view image.

For example, for an image taken from the front of the vehicle, the pixelvalues of a lower portion of this image are associated with pixels onone side of the bird's eye view image, and for an image taken from theback of the vehicle, the pixel values of a lower portion of this imageare associated with pixels on an opposite side of the bird's eye viewimage.

In step S05, a multi-lane detection method processes the bird's eye viewimage obtained in step S04, and this detection may be performed by aneural network. For example, this neural network may have a deep neuralnetwork structure (for example a convolutional neural network trainedusing annotations of bird's eye view images).

Additionally, a post-processing step S06 may be performed in which acurve is fitted on each detected lane (for example defined with curveparameters, i.e. polynomial coefficients), and a tracking step of thecurve with respect to a previously obtained plurality of images (forexample using the curve parameters).

Additionally, steps S07 and S08 may be performed using the acquiredimages of step S01 and the bird's eye view of step S04 so as to performground plane generation (step S07) and ego-motion (step S08).

For example, ground plane generation may be carried out during acalibration step, wherein the ground plane position underneath the car(in stand-still) is determined. Also, ego-motion detection may comprisedetermining the relative position of the car with respect to the groundplane, the suspension of the car allowing relative motion between thevehicle and its wheels to absorb changes and vibrations of the roadstructure. Thus, the ground plane determined during the calibration maymove with respect to the vehicle.

FIG. 2A is a schematic representation of a bird's eye view above avehicle 100, on a road where lanes L1, L2, L3, L4, L5, and L6 aremarked.

The vehicle 100 is equipped with four cameras having fields of viewvisible on the figure: 101 at the back of the vehicle, 102 at the frontof the vehicle, 103 on the left side of the vehicle and 104 on the rightside of the vehicle.

The present disclosure is however not limited to 4 cameras, more thanfour cameras may be used, for example 6 or 8 cameras.

FIG. 2B shows the four portions of each image which will be used toelaborate the bird's eye view image. Each portion is a mask and thesemasks will be assembled to form the bird's eye view of FIG. 2C. Whilethe vehicle itself is not visible on any acquired image, a graphicalrepresentation of the vehicle 105 can be placed on the bird's eye viewimage.

FIG. 3A shows four actual photographs 201, 202, 203, and 204 taken bycameras all pointing to different directions around a vehicle.

These cameras, and all the cameras that may be used to implement thepresent disclosure, may be fisheye cameras: their focal length may beless than 16 millimeters, for example less than 8 millimeters,alternatively, a fisheye camera may be defined as a camera having anangular field above 180°, for example approximatively 190°.

The present disclosure is however not limited to fisheye cameras and mayalso be implemented using other types of cameras such as pinholecameras.

FIG. 3B shows the bird's eye view image obtained with the images of FIG.3A, for example using the method of FIG. 1.

FIG. 4A shows another example of a bird's eye view while FIG. 4B showsthe detected lanes DL in a highlighted manner by implementing the methodof FIG. 1. In this illustrated example, the lanes are merely detected aslanes but there is no detection of each individual lane; for example,this detection indicates whether individual pixels of the image belongto a lane or not.

In FIG. 4C, the detection step is configured to identify each individuallane which leads to the detection of detected lanes L1, L2, L3, L4, L5,L6, and L7.

In FIG. 4D, a subsequent curve fitting is performed in a multi-curvefitting approach so that curves C1, C2, C3, C4, C5, C6, and C7 arerespectively fitted on the detected lanes L1, L2, L3, L4, L5, L6, andL7. These curves C1 to C7 may be associated with parameters such aspolynomial parameters, and a tracking may be performed over time usingthese parameters.

Also, it is also possible to use detection means such as a neuralnetwork which is trained to distinguish standard highway lanes (markedHW) and split or merge lanes (marked SM).

FIG. 5 shows a vehicle 300 comprising a system 301 configured toimplement the method of FIG. 1. The vehicle is further equipped withimage acquisition modules 302, 303A, 303B (image acquisition modules303A and 303B being located around the mirrors of the vehicle, imageacquisition module 303B being on the hidden side of the vehicle) and 304(e.g., cameras), each being oriented outwardly with respect to thevehicle and toward a different direction.

The system 301 may have the structure of a computer comprising aprocessor and a non-volatile memory including computer programinstructions to perform the method of FIG. 1.

The system 301 may therefore comprise:

-   -   a module 305 configured to elaborate a bird's eye view image of        the surroundings of the vehicle using at least one look-up-table        receiving as input the images of the plurality of images and        configured, for each image, to associate each pixel in at least        a portion of the image to a pixel position in at least a portion        of the bird's eye view image,    -   a module 306 configured to perform, on the bird's eye view        image, a detection of at least one lane marked on the surface on        which the vehicle is and visible on the bird's eye view image.

These modules may be implemented by computer instructions executed by aprocessor of the system 301.

Also, the look up tables can be stored in a non-volatile memory of thesystem 301.

Although the present disclosure has been described above with referenceto certain specific embodiments, it will be understood that the presentdisclosure is not limited by the particularities of the specificembodiments. Numerous variations, modifications and developments may bemade in the above-described embodiments within the scope of the appendedclaims.

1. A method for processing a plurality of images, each image of theplurality of images being acquired by a respective image acquisitionmodule of a vehicle and each image acquisition module being orientedoutwardly with respect to the vehicle, the method comprising:elaborating a bird's eye view image of surroundings of the vehicle usingpixel values of pixels of at least one portion of each image of theplurality of images as pixel values of the bird's eye view image, andperforming, on the bird's eye view image, a detection of at least onelane marked on a surface on which the vehicle is and visible on thebird's eye view image.
 2. The method of claim 1, wherein the detectionof lanes is performed using a neural network.
 3. The method of claim 1,wherein elaborating the bird's eye view image of the surroundings of thevehicle comprises using at least one look-up table to associate thepixels of said portions of the image of the plurality of images as pixelvalues of pixels of the bird's eye view image.
 4. The method of claim 3,wherein elaborating the bird's eye view image of the surroundings of thevehicle comprises using a look-up table for each image of the pluralityof image.
 5. The method of claim 1, comprising obtaining positioninformation of the vehicle from at least one sensor different from acamera and taking this position information into account whenelaborating the bird's eye view image.
 6. The method of claim 1,comprising performing, on the bird's eye view image, a detection of aplurality of lanes marked on the surface on which the vehicle is andvisible on the bird's eye view image.
 7. The method of claim 1,comprising a further post-processing step in which a curve is fitted oneach detected lane, and a tracking step of the curve with respect to apreviously obtained plurality of images.
 8. The method of claim 7,wherein fitting the curve on each detected lane is performed using amulti-curve fitting approach.
 9. The method of claim 1, comprising apreliminary calibration step to obtain calibration data in which pixelsof the at least one portion of each image are associated with pixels ofthe bird's eye view image.
 10. A system for processing a plurality ofimages, each image of the plurality of images having been acquired by arespective image acquisition module of a vehicle and each imageacquisition module being oriented outwardly with respect to the vehicle,the system comprising: a module configured to elaborate a bird's eyeview image of surroundings of the vehicle using pixel values of pixelsof at least one portion of each image of the plurality of images aspixel values of the bird's eye view image, and a module configured toperform, on the bird's eye view image, a detection of at least one lanemarked on a surface on which the vehicle is and visible on the bird'seye view image.
 11. A vehicle comprising the system of claim 10 andequipped with said image acquisition modules.
 12. (canceled)
 13. Anon-transitory recording medium readable by a computer and havingrecorded thereon a computer program including instructions for executinga method for processing a plurality of images, each image of theplurality of images being acquired by a respective image acquisitionmodule of a vehicle and each image acquisition module being orientedoutwardly with respect to the vehicle, the method comprising:elaborating a bird's eye view image of surroundings of the vehicle usingpixel values of pixels of at least one portion of each image of theplurality of images as pixel values of the bird's eye view image, andperforming, on the bird's eye view image, a detection of at least onelane marked on a surface on which the vehicle is and visible on thebird's eye view image.